There have been lots of opinions flying around on the internet over the last week about the forthcoming Quality Metrics that Arts Council England will be adopting as part of their assessment criteria for Arts Organisations.
One was a collection of tweets that had been aggregated into a Storify by Arts Professional. If you follow the list down to the bottom you will see that I was the very last tweet, and I spoke out in favour against an overwhelming tide of those who were anti the whole idea.
I am in favour of this for a number of reasons, firstly it allows our audiences, as well as our peers and creative teams to input into what we thought about the work. For example, the writing and the story might have been brilliant, but the lighting, and sound were poor, and the venue experience was cold and uncomfortable. These are all different aspects about an Audiences’s experience of a piece of work – An ‘AX ‘ if you like. The metrics allow us to put together a whole array of responses and create data that can be used for future planning, development and to share with other organisations, about how we might improve and develop the work we make, and for whom.
This is the world in which we now live. I am pretty sure that all the people who are being negative about this use online data all the time to make assessments and judgements. Who has never used TripAdvisor before booking somewhere? Who has never looked at an Amazon or ImDB rating before watching or purchasing a film? You can’t even buy a toaster from Tesco online without reading the reviews.
So let’s take a step back, and have a think. The Arts Council already have assessments on our work, made by a small group of assessors. In their brief, the venue, the front of House, the Programme, the Audience, as well as the work in terms of presentation and production are all asked to be assessed. So are we saying that a handful of reports are a better judgement on Artistic work then a range of responses and data from audiences across the whole tour, or lifespan of a piece of work.
What does this say about the accusation of ‘behind doors and potentially elitist judgements’ versus a range of public responses that can be combined with peer and professional assessments?
I am a director, I have used the Quality Metrics scheme, and I think the range of responses and reactions are really useful. After all, my work has been judged for years by professional critics who decide how many stars they think a show is worth. So how about now extending that for companies who don’t get reviewed in that way, and to allow those people who have actually paid for a ticket to be able to give feedback to a company.
So my questions are these, do we not think that work made by artists and companies that have received public money should not be judged by the public too? Are we not interested in the data and responses from our audiences? Do we think a behind closed doors assessment is the only true way to assess a company and their work?
The time really is to wake up and smell the coffee, and to work with this concept and to help it to develop. We are not going to go back into the analogue box, digital is here and it is up to us how we can use it and to help us to make the case for public support.
But we can only do that with support from the public…
Marcus Romer's work & blog 
I think the idea of enabling audiences to comment on the work we make, and their AX, is to be welcomed, particularly if we can find a way to generate more insightful responses than Twitter’s 140 characters allow. As a reader of Trip Advisor and other reviews, I recognize that reviews can often say as much about the reviewer as they do about the meal/B&B/product, and I pay more attention to general trends than the minutiae of one review detailing everything that went wrong. Increasingly, I find that managing audience expectation about what might happen is at least as important as what actually happens – better use of Big Data, and quality metrics available to, and generated by, audiences may help with that. Thank you for the post, Marcus.
It is how the audience are asked that is the issue for me. We use a simple google form – which does ask for a numeric ‘score’ on questions of quality etc – but the most useful one is always always the narrative box. It tells us exactly what people think, way more valuable to us than a numeric value. So we are not opposed to using digital methods to gather information at all! Its not analogue versus digital, it is how we do it….
We tried to use QM and it failed – technically (too broadband heavy to work in a rural place), and we were also unhappy that we were collecting data (email addresses) for a third party, and we didn’t feel that the questions were relevant to our work. So we went back to our tried and tested method (which is also free!).
Yes seek feedback using digital means, but no to a score card methodology of art; and there is the fear that this numeric value will ultimately be used for funding decisions. X got a higher score than Y, ergo X should receive more funding (when X was already in receipt of more funding – which may be why they received a higher score than Y) – did that audience realise they were making that comparative choice? No. Did they have any context to their ‘scoring’? No.
I am not saying this because our responses are poor, we get great feedback and ‘scores’ on the (relevant to our work) questions that we ask. Though we do get some people saying very negative things!! and we listen. But a numeric score doesn’t really give me anything… (if we have done something that didn’t work I know more by the sensuous, the fidgeting, the faces!).
The imposition of this expensive methodology by ACE is worrying to me; it uses a blunt instrument to ‘score’ and potentially compare very different things, a theatre show, an outdoor event, a participatory programme for disabled people, an education programme, an arts centre, a gallery, an experimental company.
I really don’t get how numeric data helps us to make artistic decisions over feel and intent and exploration. I would be interested to hear how others think they can. Do metrics allow us to know what people thought? how does this work exactly? How do we take scores and make decisions? I’m guessing that Van Gogh would have had very low scores in his lifetime….
whoops. wasn’t replying just to your comment Liz! didn’t scroll down far enough, thought was replying to the post in general! still…whatevs…interested in this subject discussion 🙂